Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration

In this explanatory and elaboration document Mattias Egger and colleagues provide the meaning and rationale of each checklist item on the STROBE Statement

Evaluating heterogeneity in cumulative meta-analyses

BACKGROUND: Recently developed measures such as I(2 )and H allow the evaluation of the impact of heterogeneity in conventional meta-analyses. There has been no examination of the development of heterogeneity in the context of a cumulative meta-analysis. METHODS: Cumulative meta-analyses of five smoking cessation interventions (clonidine, nicotine replacement therapy using gum and patch, physician advice and acupuncture) were used to calculate I(2 )and H. These values were plotted by year of publication, control event rate and sample size to trace the development of heterogeneity over these covariates. RESULTS: The cumulative evaluation of heterogeneity varied according to the measure of heterogeneity used and the basis of cumulation. Plots produced from the calculations revealed areas of heterogeneity useful in the consideration of potential sources for further study. CONCLUSION: The examination of heterogeneity in conjunction with summary effect estimates in a cumulative meta-analysis offered valuable insight into the evolution of variation. Such information is not available in the context of conventional meta-analysis and has the potential to lead to the development of a richer picture of the effectiveness of interventions

Can we rely on the best trial? A comparison of individual trials and systematic reviews

BACKGROUND: The ideal evidence to answer a question about the effectiveness of treatment is a systematic review. However, for many clinical questions a systematic review will not be available, or may not be up to date. One option could be to use the evidence from an individual trial to answer the question? METHODS: We assessed how often (a) the estimated effect and (b) the p-value in the most precise single trial in a meta-analysis agreed with the whole meta-analysis. For a random sample of 200 completed Cochrane Reviews (January, 2005) we identified a primary outcome and extracted: the number of trials, the statistical weight of the most precise trial, the estimate and confidence interval for both the highest weighted trial and the meta-analysis overall. We calculated the p-value for the most precise trial and meta-analysis. RESULTS: Of 200 reviews, only 132 provided a meta-analysis of 2 or more trials, with a further 35 effect estimates based on single trials. The average number of trials was 7.3, with the most precise trial contributing, on average, 51% of the statistical weight to the summary estimate from the whole meta-analysis. The estimates of effect from the most precise trial and the overall meta-analyses were highly correlated (rank correlation of 0.90).There was an 81% agreement in statistical conclusions. Results from the most precise trial were statistically significant in 60 of the 167 evaluable reviews, with 55 of the corresponding systematic reviews also being statistically significant. The five discrepant results were not strikingly different with respect to their estimates of effect, but showed considerable statistical heterogeneity between trials in these meta-analyses. However, among the 101 cases in which the most precise trial was not statistically significant, the corresponding meta-analyses yielded 31 statistically significant results. CONCLUSIONS: Single most precise trials provided similar estimates of effects to those of the meta-analyses to which they contributed, and statistically significant results are generally in agreement. However, "negative" results were less reliable, as may be expected from single underpowered trials. For systematic reviewers we suggest that: (1) key trial(s) in a review deserve greater attention (2) systematic reviewers should check agreement of the most precise trial and the meta analysis. For clinicians using trials we suggest that when a meta-analysis is not available, a focus on the most precise trial is reasonable provided it is adequately powered

Time to Update and Quantitative Changes in the Results of Cochrane Pregnancy and Childbirth Reviews

BACKGROUND: The recommended interval between updates for systematic reviews included in The Cochrane Library is 2 years. However, it is unclear whether this interval is always appropriate. Whereas excessive updating wastes time and resources, insufficient updating allows out-of-date or incomplete evidence to guide clinical decision-making. We set out to determine, for Cochrane pregnancy and childbirth reviews, the frequency of updates, factors associated with updating, and whether updating frequency was appropriate. METHODOLOGY/PRINCIPAL FINDINGS: Cochrane pregnancy and childbirth reviews published in Issue 3, 2007 of the Cochrane Database of Systematic Reviews were retrieved, and data were collected from their original and updated versions. Quantitative changes were determined for one of the primary outcomes (mortality, or the outcome of greatest clinical significance). Potential factors associated with time to update were assessed using the Cox proportional hazard model. Among the 101 reviews in our final sample, the median time before the first update was 3.3 years (95% CI 2.7-3.8). Only 32.7% had been updated within the recommended interval of 2 years. In 75.3% (76/101), a median of 3 new trials with a median of 576 additional participants were included in the updated versions. There were quantitative changes in 71% of the reviews that included new trials (54/76): the median change in effect size was 18.2%, and the median change in 95% CI width was 30.8%. Statistical significance changed in 18.5% (10/54) of these reviews, but conclusions were revised in only 3.7% (2/54). A shorter time to update was associated with the same original review team at updating. CONCLUSIONS/SIGNIFICANCE: Most reviews were updated less frequently than recommended by Cochrane policy, but few updates had revised conclusions. Prescribed time to update should be reconsidered to support improved decision-making while making efficient use of limited resources

Systematic reviews: a cross-sectional study of location and citation counts

BACKGROUND: Systematic reviews summarize all pertinent evidence on a defined health question. They help clinical scientists to direct their research and clinicians to keep updated. Our objective was to determine the extent to which systematic reviews are clustered in a large collection of clinical journals and whether review type (narrative or systematic) affects citation counts. METHODS: We used hand searches of 170 clinical journals in the fields of general internal medicine, primary medical care, nursing, and mental health to identify review articles (year 2000). We defined 'review' as any full text article that was bannered as a review, overview, or meta-analysis in the title or in a section heading, or that indicated in the text that the intention of the authors was to review or summarize the literature on a particular topic. We obtained citation counts for review articles in the five journals that published the most systematic reviews. RESULTS: 11% of the journals concentrated 80% of all systematic reviews. Impact factors were weakly correlated with the publication of systematic reviews (R(2 )= 0.075, P = 0.0035). There were more citations for systematic reviews (median 26.5, IQR 12 – 56.5) than for narrative reviews (8, 20, P <.0001 for the difference). Systematic reviews had twice as many citations as narrative reviews published in the same journal (95% confidence interval 1.5 – 2.7). CONCLUSIONS: A few clinical journals published most systematic reviews. Authors cited systematic reviews more often than narrative reviews, an indirect endorsement of the 'hierarchy of evidence'

Teaching clinical informatics to third-year medical students: negative results from two controlled trials

BACKGROUND: Prior educational interventions to increase seeking evidence by medical students have been unsuccessful. METHODS: We report two quasirandomized controlled trials to increase seeking of medical evidence by third-year medical students. In the first trial (1997–1998), we placed computers in clinical locations and taught their use in a 6-hour course. Based on negative results, we created SUMSearch(TM), an Internet site that automates searching for medical evidence by simultaneous meta-searching of MEDLINE and other sites. In the second trial (1999–2000), we taught SUMSearch's use in a 5½-hour course. Both courses were taught during the medicine clerkship. For each trial, we surveyed the entire third-year class at 6 months, after half of the students had taken the course (intervention group). The students who had not received the intervention were the control group. We measured self-report of search frequency and satisfaction with search quality and speed. RESULTS: The proportion of all students who reported searching at least weekly for medical evidence significantly increased from 19% (1997–1998) to 42% (1999–2000). The proportion of all students who were satisfied with their search results increased significantly between study years. However, in neither study year did the interventions increase searching or satisfaction with results. Satisfaction with the speed of searching was 27% in 1999–2000. This did not increase between studies years and was not changed by the interventions. CONCLUSION: None of our interventions affected searching habits. Even with automated searching, students report low satisfaction with search speed. We are concerned that students using current strategies for seeking medical evidence will be less likely to seek and appraise original studies when they enter medical practice and have less time

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews

SUPPORT Tools for evidence-informed health Policymaking (STP) 8: Deciding how much confidence to place in a systematic review

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers

Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali's essential medicines

BACKGROUND: Research findings are increasingly being recognized as an important input in the formation of health policy. There is concern that research findings are not being utilized by health policy-makers to the extent that they could be. The factors influencing the utilization of various types of research by health policy-makers are beginning to emerge in the literature, however there is still little known about these factors in developing countries. The object of this study was to explore these factors by examining the policy-making process for a pharmaceutical policy common in developing countries; an essential medicines list. METHODS: A study of the selection and updating of Mali's national essential medicines list was undertaken using qualitative methods. In-depth semi-structured interviews and a natural group discussion were held with national policy-makers, most specifically members of the national commission that selects and updates the country's list. The resulting text was analyzed using a phenomenological approach. A document analysis was also performed. RESULTS: Several factors emerged from the textual data that appear to be influencing the utilization of health research findings for these policy-makers. These factors include: access to information, relevance of the research, use of research perceived as a time consuming process, trust in the research, authority of those who presented their view, competency in research methods, priority of research in the policy process, and accountability. CONCLUSION: Improving the transfer of research to policy will require effort on the part of researchers, policy-makers, and third parties. This will include: collaboration between researchers and policy-makers, increased production and dissemination of relevant and useful research, and continued and improved technical support from networks and multi-national organizations. Policy-makers from developing countries will then be better equipped to make informed decisions concerning their health policy issues